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APPLYING 

COSTS, RISKS, AND VALUES EVALUATION (CRAVE) METHODOLOGY 

TO * 

ENGINEERING SUPPORT REQUEST (ESR) PRIORITIZATION 

Executive Summary 

Given limited budget, the problem of prioritization among Engineering Support Requests 
(ESRs) with varied sizes, shapes, and colors is a difficult one. At the Kennedy Space 
Center (KSC), the recently developed 4-Matrix (4-M) method represents a step in the 
right direction as it attempts to combine the traditional criteria of technical merits only 
with the new concern for cost-effectiveness. However, the 4-M method was not 
adequately successful in the actual prioritization of ESRs for the fiscal year, 1995 
(FY95). This research identifies a number of design issues that should help us develop 
better methods. It emphasizes that given the variety and diversity of ESRs one should not 
expect that a single method could help in the assessment of all ESRs. One conclusion is 
that a methodology such as Costs, Risks, and Values Evaluation (CRAVE) should be 
adopted. It also is clear that the development of methods such as 4-M requires input not 
only from engineers with technical expertise in ESRs but also from personnel with 
adequate background in the theory and practice of cost-effectiveness analysis. 

At KSC, ESR prioritization is one part of the Ground Support Working Teams (GSWT) 
Integration Process. It was discovered that the more important barriers to the 
incorporation of cost-effectiveness considerations in ESR prioritization lie in this 
process. The culture of integration, and the corresponding structure of review by a 
committee of peers, is not conducive to the analysis and confrontation necessary in the 
assessment and prioritization of ESRs. Without assistance from appropriately trained 
analyses) charged with the responsibility to analyze and be confrontational about each 
ESR, the GSWT steering committee will continue to make its decisions based on 
incomplete understanding, inconsistent numbers, and at times, colored facts. The current 
orga ni zational separation of the prioritization and the funding processes is also identified 
as an important barrier to the pursuit of cost-effectiveness. Perhaps the greatest barrier is 
that, at the working level, KSC’s culture is so preoccupied with technical concerns that it 
seems almost oblivious to any cost concerns, let alone cost-effectiveness concerns. It is 
recommended that we must urgently begin to change that culture and seek a better 
balance between these two concerns. 
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The 4-Matrix method 
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Analytic Hierarchy Process 
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1. BACKGROUND 


1 


“Our team pledges the highest level of performance at the lowest possible 
cost within the framework of absolute dedication to safety and quality to 
meet the National Space Initiatives and challenges of today and the 21st 
Century.” 

From p.3 of the March 24. 1994 document. 
Implementation of the KSC 1994 Strategic Management Plan 


To keep this pledge, at KSC, decisions such as which engineering support requests 
(ESRs) to fund and at what level must be based on sound cost-benefit / cost-effectiveness 
analyses (CBA). Toraskar and Joglekar [1993] have shown that although CBA theory 
contains rich concepts and sound principles, its practice is often based on overly 
simplified and narrow techniques supplemented with numerous convenient but incorrect 
assumptions. Consequently, often the practice of CBA is trapped in the type of vicious 
circle [See Exhibit 1] I observed last year, here at KSC, in the context of advanced 
software decision-making [Joglekar, 1993]. 

In order to break out of such vicious circles, we must use a methodology such as Costs, 
Risks, and Values Evaluation (CRAVE). A methodology is a way of thinking — armed 
with cer tain fundamental principles and concepts — and a way of analyzing available 
decision alternatives using situationally appropriate techniques and tools from a large 
body of methods consistent with the fundamental principles and concepts. A 
methodology focuses on both the techniques and the processes used in relevant decision- 
making. Joglekar and Toraskar [1994] have explained exactly what CRAVE is, and how 
it is supposed to work. Exhibit 2 captures Joglekar and Toraskar’ s prescription for the 
process of applying CRAVE written in the context of global technology deployment 
(GTD) decisions. The six-stage process emphasizes some of the fundamentals of 
CRAVE, such as 

• the need for context articulation, 

• the recognition that cost-benefit estimation is in itself a costly process and 
sometimes the benefits of estimation do not justify the costs, 

• the recognition that it is not necessary to force quantification on certain 
consequences that are truly non-quantifiable 

• the recognition that measurement and valuation should not be confused as one 
and the same; valuation of a consequence is in the eye of the beholder, and 
hence a multi- perspective analysis is more desirable than a single perspective 
analysis. 
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2 


As Exhibit 3 shows, CRAVE retains some of the fundamental principles and .looks 
forward to the use of many rich typologies and concepts of classical CBA. Yet, CRAVE 
attempts to overcome the typical flaws in the practice of traditional CBA. Exhibit 4 
shows how CRAVE differs from classical CBA practice. [For a fuller understanding of 
CRAVE, see Joglekar and Toraskar, 1994]. 
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2. THE TASK THIS YEAR 

This year. Bob Lang, Director of Vehicle Engineering, asked me to 

(a) See how CRAVE could be applied to the problem of ESR prioritization, and 

(b) Recommend a method for future use. 

Given that task, I tried to understand: 

1. What exactly an ESR is, how varied ESRs are from one another, and what 
kinds of peculiarities characterize some of the ESRs; 

2. What explicit and implicit criteria are currendy used in the prioritization of 

ESRs; and 

3. What process is currendy used in the prioritization of ESRs. 

Clearly, my understanding of these three topics was not a linear but a parallel process, 
with the understanding of one topic influencing the understanding of other topics, and 
vice versa. It follows that readers who want to truly grasp my total understanding may 
want to read this report and its appendices at least twice in its entirety. 

Briefly, an ESR is a proposal to upgrade an equipment or a facility used in the “ground 
processing” (all the work done between one landing and the next launch) of a space 
shutde. An ESR seeks one or more of desired goals such as improving flight safety, 
improving safety of ground processing operations, overcoming system obsolescence, 
avoiding schedule delays, and reducing processing costs. At KSC, there are over thirty 
engineering systems (represented by their GSWT teams) that are together seeking 
funding for some 200 different ESRs, each costing anywhere between a few thousand to 
a few million dollars. Given limited resources, only a few of these ESRs can be fully or 
partly funded in any given year. 

The benefits sought by some ESRs are considered so critical to the strategic objectives of 
KSC that these ESRs are designated as Category 1 (i.e., mandatory), and they are 
implemented expeditiously. This research is focused on the prioritization of Category 2 
(i.e., desirable but optional) ESRs, where cost-effectiveness analysis ought to be useful. 
However, as Appendix A explains, historically, even Category 2 ESRs were prioritized 
by using a system called “P-cut rating,” which extended the notion of “required” versus 
“desirable” within Category 2. Cost-effectiveness considerations were simply left out. 
(See attachments A-l and A-2). 

Therefore, I was pleasantly surprised to know that, to incorporate cost-effectiveness 
considerations in the prioritization of Category 2 ESRs, recently, GSWT steering 
committee had developed a new approach, named “the 4-Matrix (4-M) method.’ (See 
Attachment A-3). Clearly, the 4-M method represents a step in the right direcnon, and 
its designers must be applauded for their courage in breaking away from the tradinon of 
focusing exclusively on the technical merits of an ESR. 


270 


4 


On the other hand, it seems that the designers had no help from anyone with the proper 
background in the theory and practice of cost-effectiveness analysis. Consequently, the 
design shows numerous flaws detailed in Appendix A, including: 

1. The 4-M method ignores two over-riding considerations at KSC, namely 
safety and obsolescence. 

2. The 4-M method actually focuses on two primary factors: 

(a) Expected annual impact on flow schedule, and 

(b) Payback period. 

3. A multiplicative model is inappropriate for the 4-M method. 

4. The design of the rating categories and score ranges is inappropriate. 

5. The absence of relevant probability considerations on the Cost Assessment 
Worksheet is inappropriate. 

6. The language is ambiguous, and instructions / explanations are lacking. 

As a consequence of these design flaws, the 4-M method was not successful in assisting 
the prioritization of FY95 ESRs. Table 1 and Figures 1, 2, and 3 confirm this lack of 
success by showing that there is no relationship between the steering committee’s actual 
priorities and either the cost or the payback or the ESR scores. 

These and other design flaws are also manifest in the assessments of specific ESRs 
reviewed in Appendix B. As can be seen, the design of 4-M leads to a variety of 
instances of misrepresentations and overestimation of the benefits. At the same time, 
there are instances of ESRs for which 4-M worksheets are very difficult to use. Some of 
the comments in Appendix B further show that we seem to aggravate the problem of 
comparing ESRs by lumping together ESRs costing $4M with those costing $4K. Figure 
4 graphically shows how a single one of the top 17 ESRs accounts for over 66% of the 
funding needed by all 17 put together. Clearly, ESRs requiring capital expenditures 
ought to be evaluated and funded separately from ESRs requiring a few thousand dollars. 

Most importantly, the variety of sizes, shapes, and colors of ESRs captured by Appendix 
B makes it clear that the presumption of being able to create a single method to fit all 
ESRs is unrealistic. What we need is a methodology, and not a method. A methodology 
allows for the use of several different methods of analysis depending on the nature of the 
technical system, the type of modification sought, and the relevant costs and benefits. 
Despite the shortcomings of the 4-M method, we must not abandon the pursuit of cost- 
benefit analysis, or return to the arbitrary decision making of the past Instead, we must 
evolve towards the adoption of a methodology such as Costs, Risks, and Values 
Evaluation (CRAVE). 

In any case, the most important reasons for the lack of success of the 4-M method lie in 
the GSWT integration process. The latter part of Appendix A discusses these process 
issues in depth. Here, by way of conclusion, I simply want to list several process related 
barriers to the application of CRAVE, and my recommendations to management for 
overcoming those barriers. 
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BARRIERS AND CORRESPONDING RECOMMENDATIONS 


Barrier 1. 

We are trying to develop a single method to assess all ESRs. 

However, ESRs come in a variety of sizes, shapes and colors. 

Their assessment calls for a methodology - a collection of methods, along with 

the knowledge of which one to use when. 


Recommendation : 

Adopt CRAVE. 


Barrier 2. 

We leave the development of a cost-benefit analysis (CBA) method to engineers 
who do not have the necessary background in the theory and practice of CBA. 

Recommendations: 

Get expert help. 

Develop internal expertise through training. 


Barrier 3. 

The culture of “integration” (and the corresponding structure of review by a 
committee of peers) is not conducive to the analysis and confrontation necessary 
for objective assessment of each ESR, and the subsequent rational prioritization 
among them. 

This is true whether the assessment criteria include cost considerations or not 

Recommendation: 

Let the GSWT steering committee be assisted by appropriately trained, objective 
analysts, charged with the responsibility to analyze and be confrontational about 
each ESR. 
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Barrier 4. 

Organizational separation of the prioritization, funding, and implementation 
processes is not conducive to the pursuit of cost-effectiveness. 

Recommendation : 

To the extent possible, centralize the authority to prioritize, fund, and monitor 
implementation of ESRs. 

At least, coordinate sufficiently to make sure that decisions in these three 
processes are based on the same criteria, and on an appreciation of each others’ 
reasoning and deliberations. 


Barrier 5 (The Most Important One). 


At the working level, KSC’s culture is almost oblivious to any cost concerns, let 
alone cost-effectiveness concerns. 

Recommendation : 

Begin to change the culture by 

• insisting on a discussion of costs, and paybacks at meetings to review priority 
list, and at funding decision meetings, 

• tracking and monitoring the estimated and actual costs of ESRs approved in 
prior years, 

• setting up a program of training in CBA / CRAVE concepts and principles. 
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Exhibit 1 
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The VIciau3 Circle 


Available C3A studies measure only selecad 
(not ail) changes brought about by the development 
and implementation at a given ASW. At times, they 
fores qualification of the non-quantifiaDie, or prediction 
or the unpredictable. The baselines used in the 
measurement are often incorrect. Measurement (In 
onysical units) is not separated from valuation. 
Valuation is from a single (as against each 
stakeholders separate) point of Mew. Values are not 
adjusted for their probabilities or timing of occurrence. 
Sensitivity analysis is not done. In short, many 
principles of the C3A methodology are violated 


T 


The- focus of the CSA studies is 
primarily on die quantifiable. Very 
important but non-measurabie_cost. 
benefits, and risks ara left out. i ne 
margin of errors in the Quantified 
estimates is very large. The real values 
of me Agency mission, me values or the 
senior managers, me values of me 
contractors, etc., are not captured by 
me analysis. 
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Feedback and Updates 


Exhibit 2 


A Procsss for Applying CRAVE Methodology to GTD Decisions 


Stage 1; Contest Articulation 

Identify available cedsion alternatives. 
Identify all stakeholders 
Guesstimate upper and lower bounds 


Define horizon. 
Us; assumoticns 


^ Conduct ^ 
"ormaJ CRAVE? 


Record the reasons far 
future reference 


T VS 

Stage 2 ; Enumeration of 

these changes due to new technology 

- Use of resources, 

• Productivity, 

* Profitability. 

- Strategic-fit. and 

- Risks (technical, schedule, operational) 
during anc after technology deployment. 


in all pertinent organizations 
- Sponsoring deoartment, 

- E&O function, 

- foreign sucsidiary. 

- Parent corporation, 

- Host country 



Stage 2 : Measurement Issues 

- PropeA basa-imes. 

- Joint use of res curcas, 

- Cc-orccucars 


Provide 

a clear and complete 
description 


be vajuea^ 
^cuantitaxiveiy? 
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Some Commonalties in CBA and CRAVE Methodologies 

Fundamental principles such as: 

• Formal and explicit analysis 

• Accounting for ail incremental costs and benefits 

. Accounting for costs and benefits to whomsoever they accrue 

• Adjusting for time value of various costs and benefits 

• Explicit recognition of assumptions, and performance 

of sensitivity analysis for alternative assumptions 

Use of typologies of costs and benefits such as: 

• Fixed and variable. 

• Direct and indirect, 

• Obvious and hidden. 

. Primary, Secondary, tertiary 

• One-time and recurrent 

• Controllable and non-controllabie 

Use of rich concepts such as: 

• Opportunity costs 

• Pareto-superiority 

. Joint use of resources 

. Cause-effect versus multi-producers-singie-product relationships 


OWQINAL page It 

pooh QUAtmr 
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Exhibit 4 


\ COMPARISON OF CURRENT CBA PRACTICE 
WITH CRAVE METHODOLOGY 


CBA PRACTICE 


CRAVE METHODOLOGY 


Justification focus 

Retrospective 

T echnique-oriented 

Single-perspective assessment 

Consolidation of muitipie stakeholders' 
criteria (possibly AHP type) 

Forced quantification of the 
non-quantifiable 

Left to the proponents of specific GTDs 

Left to non-experts with certain rectoes 

Potential for advocacy ignored 

Economic values only 

Risks of non-existence of co-producers 
ignored 

Consideration of internal risks oniv 

Stringent financial criteria 

Short-sighted perspective 

Product-oriented (i.e. C/B ratio, or IRR) 

Zero valuation of non-quantified 
intangibles 

Measurement and valuation 

confused with one another 

Evaluating given alternatives 

Often unrealistically long horizons 

Ad hoc methods and assumptions 


Decision Assistance focus 

Prospective 

It is a methodology 

Multi-perspective assessment 

No simple consolidation 

(concept of Pareto superiority) 

Quantification oniy when possible 

Actively directed by the MNC managers / 
decision makers 

Guided by a methodologist 

Potential for advocacy actively managed 

Economic and non-economic values 

Risks of non-existence of co-producers 
explicitly accounted for 

Consideration of both internal and external risks 

More realistic criteria 

Balanced perspective 

Process-oriented 

Explicit consideration of non-quantified 
intangibles 

Measurement separated from valuation 

Helping construction ofbetter alternatives 
Weil-defined, reasonably long but limited horizons 
Consistent methodology and assumptions 
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Costs of the ESR 


FIGURE 1 


ESR Costs and Priorities 
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Appendix A 

A Critique of the 4-Matrix (4-M) Method of ESR Assessment 


by 

Praf Joglekar 


The Ground Systems Working Teams (GSWT) Steering Committee has a difficult task: 
Prioritization among hundreds of worthy Engineering Support Requests (ESR). These 
ESRs come from some 31 different vehicle, payload, and facility systems. Individual ESR s 
cost anywhere between a few thousand to a few million dollars, and seek one or more of 
KSC’s goals of improving safety, avoiding delays in flow-cntical activities, reducing 
obsolescence, and reducing operations and maintenance (O&M) costs. At the same time, 
an important concern at KSC is that the modifications done by one engineering system 
must not adversely affect the functioning of other engineering systems vital to the 
processing of the shuttle. Thus, the process of ESR prioritization is actually called, 
“GSWT Integration.” Attachment A-l describes this process and its ground rules as 
of August 1993. 


To understand the first ground rule, it should be realized that there is a system of 
designating ESRs as either Category 1 (i.e., mandatory) or Category 2 (i.e., highly 
desirable, but not mandatory). Suffixes such as S, M, or E are added to these categories to 
indicate whether ESR attempts to address a safety, or management, or environmental 
requirement The ultimate authority for all ESR category designation rests with the 
Ground Review Board (GRB). The funding and approval of Category 1 ESRs is outside 
the GSWT process, which focuses on prioritizing within Category 2 ESRs. 


To compare ESRs with one another and prioritize them is similar to comparing and rank- 
ordering hundreds of apples, oranges, bananas, strawberries, and watermelons, fruit by 
fruit! Each fruit comes in a variety of sizes, shapes, colors, and ripening stages. To 
complicate the matter further, the fruits come in a variety of semi-transparent packages 
designed to exaggerate their attractiveness. Some fruits sell by the pound, other by the 
piece, still other by the package. For some quantity discounts are available, for some 
others their are cost-premiums associated with the timing of purchase. We want to 
maximize the nutritional value, the taste, the flavor and still be within our budget, which 
will be determined at future dates by several different funding authorities! 

As I understand it, in the past, this comparison and rank-ordering was not a major 
problem. We had resources to buy plenty of each kind of fruit. Even ESRs that were not 
funded in a given year were seen as simply deferred for implementation in a year or two. 
Thus, as described in Attachment A-l, we prioritized ESRs on the basis of their technical 
merit (perhaps a euphemism for subjective judgment based on such inputs as which team 
makes the best presentation, who is championing an ESR, which team is screaming the 
loudest, etc.), with practically no regard to what they cost. 
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To assist the steering committee in its assessment of the technical merits, GSWT teams 
were encouraged to assign a “P-cut” raring to individual ESRs.. Attachment A-2 presents 
the P-cut definitions. Although, the GSWT steering committee is charged to consider only 
Category 2 (desirable but not mandatory) ESRs, the P-cut system extended the concept of 
mandatory versus desirable nature of ESRs to several levels within Category 2 by focusing 
on the reasons underlying the modifications sought by. the ESRs, and within Category 2 
ESRs getting PI, P2, or P3 ratings were seen as required!. The differences in the costs of 
various ESRs were not seen as an important consideration, and even highly cost-efficient 
ESRs (those promising a payback in less than one year) received a low priority of P4-2, 
practically ensuring that we will never get around to these ESRs with the limited available 
funds. 

Over the last three years, the resources available for NASA, in general, and for ESRs in 
particular have shrunk dramatically. Now, we must be cost-conscious and choose between 
competing ESRs systematically and rationally. In other words, today we must base our 
decisions not only on technical merit but also on cost-effectiveness considerations. 
Recognizing this need for cost-effectiveness considerations, the GSWT Steering 
Committee recently developed and implemented a method called the 4-Matrix (4-M) 
method of ESR assessment. See Attachment A-3 for the description of the method, 
the forms used, and the accompanying instructions. 

The designers of 4-M method must be applauded for their courage in breaking away 
from the established culture at KSC which is oblivious to any cost considerations, let 
alone consideration of cost-effectiveness. 4-M represents a first attempt at bringing a 
degree of rationality to the process of rank-ordering ESRs. This method must evolve and 
lead to a sounder methodology in the future. The critique here is intended to assist in that 
evolution. It must not undermine the credit for a pioneering effort on the part of the 
designers of 4-M. 

I had the opportunity to study how the 4-M method worked in the steering committee’s 
rank-ordering of the top 17 Vehicle Engineering (TV) related ESRs for FY95. Table 1 
summarizes the data on cost, payback period, and ESR score, for each of the top 17 
ESRs. In addition, I have included the Category / P-cut rating, where available. Finally, I 
was told that some of these top 17 ESRs were originally in Category 1 but were 
downgraded to Category 2. The Column “Old Category” reports this information. 

As can be seen from Table 1, an ESR’ s priority does not seem to be related to either the 
size of the project (as defined by its cost), or its payback period, or the ESR score. 
Figures 1, 2, and 3 graphically demonstrate the lack of these relationships. In short, the 
steering committee’s priority order cannot be explained by any of these factors. It follows 
that the 4-M method does not seem to help in the rank-ordering of ESRs. There are 
several reasons why. Here, these reasons are grouped in two major areas: issues pertaining 
to the design of the 4-M method, and those pertaining to the GSWT process. 
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DESIGN ISSUES: 

1. The 4-M method ignores two over-riding considerations at KSC, namely 
SAFETY and OBSOLESCENCE. 

There is no factor to assess the severity of the consequences of potential mishaps caused 
by the existing equipment or facilities in terms of injury to personnel, or damage to the 
orbiter or the payloads. Nor is there a factor to assess the obsolescence and lack of 
supportability of the existing equipment or facilities. Thus, in addition to the ESR scores, 
the GSWT Steering Committee has to consider these factors. Since at KSC, safety is 
indeed our Number 1 concern, it is natural that we give a high priority to safety related 
ESRs regardless of their 4-M scores. The “category” and “old category” information in 
Table 1 confirms this. ESRs that had an Old Category 1, and those with 2-s designation 
seem to be at the top of the list of 17 regardless of their costs, ESR scores, or payback 
periods. 

2. The 4-M method actually focuses on two primary factors: 

(a) Expected annual impact on flow schedule, and 

(b) Payback period. 

Note that three of the four factors in the 4-M method, namely 

i. Frequency of Operation 

ii. Processing Impact/ Improvement 

iii. Likelihood of Occurrence 

when multiplied together, basically give us a score for the expected annual impact on 
flow schedule. 

The payback period is calculated separately in the Cost Assessment Worksheet 

3. A multiplicative model is inappropriate for the 4-M method. 

In theory, if all incremental costs, risks, and benefits can be identified, measured, and 
valued in dollar terms, the calculated Payback period should be the sole criterion for 
rank-ordering ESRs. The shorter the Payback, the more desirable the ESR. 

However, often it is difficult, if not impossible, to measure and value in dollar terms 
factors such as Safety Improvement, Obsolescence Reduction, and Avoidance of Schedule 
Delays. In that case, it makes sense to use a factor scoring and weighting method. But 
the scores on such factors should not be multiplied with each other or with the score on 
Payback. What we need is a weighted average of the scores. The weights have to be 
carefully chosen to reflect the values of the organization. A method called the Analytical 
Hierarchy Process (AHP) is most helpful in the development of proper weights, and 
KSC should explore its application. 
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4. The design of the rating categories and score ranges is inappropriate. 

4-M’s multiplicative approach is particularly inappropriate since, by design, the score for 
Expected Schedule Impact can range from 1 to 100 while the score for Payback can range 
only from 1 to 5. Thus, 4-M is biased towards minimizing the importance of Payback. 
This bias is farther accentuated by the fact that all ESRs providing a longer than 2-year 
payback (including those that may never payback) receive a rating of 1, while ESRs 
providing a payback within 3 months receive a rating of only 5. I believe this is totally 
inappropriate. 

The way I see it, provided the payback calculations are reasonably valid and 
accurate, all ESRs promising a payback within one year deserve to be immediately 
funded. That will have more money (or other resources) at the end of the year than we 
had at the beginning, and those increased resources would be available for the pursuit of 
our other goals such as increased safety. Even ESRs promising a payback within 4 years 
are highly desirable. They represent better than 20% per year compounded rate of return. 
On the other hand ESRs that take longer than 5 years to payback should be considered 
undesirable unless they promise significant improvements on one or more of the non- 
quantifiable dimensions of safety, obsolescence, or schedule. 

Thus, I believe that 4-M’s design of the rating categories as well as possible range of 
scores for the Payback factor are inappropriate. 

An important attribute of properly designed rating categories is that they are mutually 
exclusive (i.e., non-overlapping) and collectively exhaustive (i.e., no possibility should be 
left out). Unfortunately, in 4-M we see both of these attributes violated: 

• The critical path (i.e., serial flow) nature of an operation seems to be double-counted, 
once in the Frequency of Operation factor, and again in the Processing Impact/ 

Improvement factor. 

• The categories in the Likelihood of Occurrence factor provide no room for the rating 
of a mishap that is expected to happen no earlier than 18 months and no later than 24 
months. Thus, the categories here are not collectively exhaustive. 

• Given that we have already accounted for the frequency of an operation, the likelihood 
of occurrence factor should refer to likelihood per operation , rather than like lihood 
per year. This is a second example of double-counting certain factors. 

5. The absence of relevant probability considerations on the Cost Assessment 
Worksheet is inappropriate. 

Many ESRs are designed to avoid potential mishaps likely to be caused by existing 
equipment or facilities, and hence will contribute their estimated savings in “annual 
corrective manhours and materials” only with the probability of occurrence of the mishap. 
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The expected savings are the product of this probability times the estimated savings. Yet, 
the Cost Assessment Worksheet does not require an estimation of the relevant probability. 
Consequendy, the computation implicitly assumes that the mishap would occur with 
certainty. The net result is that typically, 4-M’s Cost Assessment Worksheet 
underestimates the payback period. (For an example, see comments on kl5836 in 
Appendix B). 

6. The language is ambiguous, and instructions / explanations are lacking. 

One well-known short-coming of cost-benefit analysis is that it can easily become an 
instrument of intentional or unintentional biases introduced by the champions and 
advocates of specific projects. Therefore, in designing a cost-benefit assessment method, 
one has to be extra careful and try to minimize opportunities for the introduction of such 
biases, and convenient misinterpretations of the words and phrases chosen. Unfortunately, 
4-M is fertile with many opportunities for the introduction of such biases: 

• One factor is defined simply as " Processing Impact / Improvement ,” without any 
further instruction. The designers of 4-M assume that there will never be an ESR that 
will actually negatively impact the flow schedule. But the fact is, in some cases for the 
sake of safety, we may introduce new procedures that could actually delay a shuttle’s 
serial flow. For such an ESR, the designers of 4-M would like to see that the lowest 
rating of 1 is used. However, the lead engineer who knows that the concerned 
operation is a serial one, refuses to use a 1 or a 2. Indeed, one cannot blame a lead 
engineer whose ESR introduces a 24-hour serial delay if he/she chose to rate the ESR 
as a 5. 

• 4-M asks that an operation’s frequency be rated either as a “5 Multiple times per flow 
/ month,” or as a “4 Once per flow / month.” Given the ambiguity in the phrase “per 
flow / month,” we just cannot blame a champion, whose ESR truly deserves a “4,” if 
he/she rates it a “5.” After all, if the operation is once a month, then it is clearly 
multiple times per flow since a flow lasts for several months. If the operation is once 
per flow, given the overlapping nature of several shuttle flows, in many months in a 
year the operation may be done multiple times a month! 

• The designers of 4-M use the phrase “likelihood of occurrence” to refer to the 
“probability that a mishap or ‘problem condition’ would occur unless an ESR was 
implemented'. Yet, several lead engineers I have talked to assume it to mean the 
“ probability that the operation will be done within the next so many flows." (See 
comments on kI58I3 in Appendix B). 

• Among those who realize that “likelihood of occurrence” referred to the probability of 
a mishap, many say that faced with the choice between rating their ESR a “3 Could 
happen within next 4 flows / 6 months” or a “2 Could happen within next 8 flows / 12 
months,” they invariably choose a “3” rather than a “2.” As one lead engineer puts it. 
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PROCESS ISSUES: 

1. KSC culture is not cost-conscious. 

The greatest single barrier to the implementation of any method of cost-effectiveness 
analysis at KSC is KSC’s culture which is almost oblivious to any cost concerns, let 
alone cost-effectiveness concerns. In the GSWT process this lack of cost-consciousness 
is evident in many ways: 

• As Table 1 and my Appendix B comments on kl4453 show, the GSWT process does 
not distinguish between a $4K ESR and a $4M ESR. 

• Note from Table 1 that the steering committee has included U5825 in its list of Top 
17 ESRs even though there is no rough order of magnitude (ROM) estimate of its 
cost. In other words, the ESR could cost anywhere between a few thousand to a few 
million dollars, and it would still be Priority #10 on the list 

• At the Thursday morning meetings of the steering committee, often a report is given 
on the status of ESRs approved in prior years. While that report includes information 
on which ESRs are complete, which ones are behind schedule and by how much, etc., 
there is no information on the original cost estimates and actual expenditures on 
the various ESRs. In fact, I am told that the way our accounting system is set up, it 
just cannot track and monitor ESR by ESR expenses. 

• At the June 28, 1994 meeting to review TV’s FY 95 priority list, management did 
not raise any questions about costs, paybacks, or the 4-M scores of any of the 
ESRs. The steering committee also did not make any attempt to defend its priority list 
using any one of these cost-related considerations. 

Given this culture, most engineers do not take ESR cost estimates seriously. Even 

those who are charged with arriving at the estimates freely admit that their estimates 

could be wrong by 50-60%. Others put the expected margins of errors to be as high as 

80-100%! Given that, for many ESRs, benefits are inherently more difficult to quantify 

and measure, sometimes we even see instances of carelessly exaggerated claims. (See 

Appendix B comments on kll794). Others see no need to even attempt to quantify the 

benefits of their ESRs, or report any 4-M scores. They know very well that as long as / 

their ESR can claim to be a “mission-stopper,” it will be funded regardless of costs. Some 

team members speculated that the 4-M method creates the paperwork necessary to 

pretend to Washington DC that we are cost-conscious. 

Thus, I was not surprised that many teams had not read 4-M instructions carefully, and 
made such elementary mistakes as addition in place of multiplication, or reporting payback 
in years as payback in months (See Appendix B comments on k!5040). 
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2. Organizational separation of the prioritization, funding, and implementation 
processes is not conducive to the pursuit of cost-effectiveness. 


The second most important reason why any method of cost-effectiveness analysis would 
be very difficult to implement in the GSWT process is the organizational separation of the 
process that does the prioritization from the process that provides the funding, which in 
turn, is separated from the process that ensures actual implementation of an ESR. While 
GSWT teams and steering committee are asked to prioritize the ESRs, they have no 
authority to actually fund or implement any ESR. The consequences of this separation 

are: 

• It perpetuates the culture of a lack of cost-consciousness, and GSWT teams and 
steering committee are likely to continue to focus exclusively on technical merit and 
ignore all cost considerations. 

• It causes considerable frustration for the teams and the steering committee 
insofar as the funding mechanism can override the priorities set by the GSWT 
integration process, and the implementation mechanism (i.e.. Design Engineering [DE] 
and Shop Floor [SF] ) can easily re-arrange those priorities by allocating or not 
allocating the necessary manpower to specific ESRs. Furthermore, even if the steering 
committee wants to use cost-effectiveness as an important criterion, given that our 
long standing culture is so deeply ingrained throughout KSC, DE and/or SF may 
continue to use technical merit as the sole criterion in deciding which ESRs to work 
first, if at all. 

• Both, the total manpower spent, and the calendar time needed, for all this 
prioritization, funding, and implementation is inordinately high. In other words, the 

bureaucracy is perpetuated. 

3. The culture of integration is not conducive to the analysis and confrontation 
necessary in a rational prioritization among competing investment 
alternatives. / 

At KSC, the ESR prioritization process is called “GSWT integration process,” and it is 
left to a committee consisting of representatives of several GSWT teams (i.e., a committee 
of peers). This is consistent with KSC’s long standing tradition of decision-making by 
consensus developing teams. The GSWT integration process should help ensure that one 
team’s ESR does not interfere with the functioning of other team’s systems. However, it 
may not be the most productive process for prioritization among competing demands on 
limited resources. Appendix B clearly shows that for a rational prioritization of ESRs, we 
need considerable analysis and confrontation. 

As members of “an integration process,” individuals on the steering committee are inclined 
to respect the expertise and trust the integrity and judgment of individual teams_ For 
example, from Attachment A-l, page 2, note that a team’s internal priorities of its ESRs- 
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are not to be questioned by the steering committee. Yet, as is clear from Appendix B, 
these teams have no expertise in cost-benefit estimation. Consequently, a team’s cost- 
benefit estimates may be way off the mark, and its internal priorities may be based on such 
erroneous estimates. On the other hand, GSWT teams do have substantial vested interests 
in making their ESR look as attractive as possible. Consequently, a team may be 
misrepresenting the true benefits (see Appendix B comments on kl5040), or 
overestimating the magnitude of those benefits (see Appendix B comments on k!5813, 
U1794, and kl 5836). 

4. The need for a properly trained and impartial analyst to assist the steering 
committee. 

Of course, even if we renamed the GSWT Steering Committee as “ESR Prioritization 
Committee,” we will not get the necessary degree of analysis and confrontation from this 
committee, for several reasons: 

• First, most members of the committee do not have the necessary background or 
training to do the kind of analysis I have done. 

• Second, given that 

a) serving on the steering committee is only one tenth of one’s job, and 

b) the number of ESRs to consider is very large 
(48 were considered in prioritizing the TV list), 

no one has the time to check the claims, the assumptions, and the computational 
accuracy of each one of the ESRs. 

• Third, in a structured team, members of the team could be assigned to check out 
specific ESRs, ensuring that each ESR is checked by one or more team members. In a 
committee of peers, such a division of labor is very difficult 

• Finally, a committee of peers thrives on the collegiality of its members. Such a 
committee hates confrontational members, and no individual is likely to volunteer to be 
a confrontational member. 

Thus, if the GSWT steering committee truly wants to use cost-effectiveness as a criterion 
in its prioritization of ESRs, it must seek assistance from a properly trained, impartial 
analyst who is charged to be confrontational about every ESR, its logic, cost- benefit 
estimates, computational accuracy, etc. 
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“Almost anything that could happen over the next 12 months, could also happen over 
the next 6 months, except perhaps an increase in my salary.” 

• Although the instructions are to multiply the scores on the four matrices to obtain the 
total score, the use of the word “total” seems to have led some teams to add the 
scores on the four matrices. (See comments on l U5813 in Appendix B). 

7. The presumption of being able to create a single method to fit all ESRs is 
unrealistic. 

When we force different discipline engineers to all use particular forms, and a given set of 
cost categories, we soon discover that our method does not quite fit all the systems we are 
trying to evaluate. There are many tell-tale signs that the 4-M method does not fit all 
ESRs. 

• 7 (15%) of the top 48 ESRs for FY95 were submitted without an attempt to fill out 
either of the two forms of the 4-M method. 

• 6 (13%) more filled out the ESR score form, but not the Cost Assessment Worksheet 

• Regardless, 3 of these “defiant” 13 wound up on the Steering Committee’s list of the 
top 17 ESRs. 

• Those who did fill in the forms, often had to modify the cost assessment worksheet to 
report the some of the unique costs or benefits of their system. 

• For more specific examples of the inapplicability of 4-M method, see Appendix B. 

The variety of technical disciplines the ESRs originate in, the diversity of benefits they 
seek, the variety of risks they reduce, the many different types of costs they impose, 
together suggest that to evaluate and rank-order these ESRs, what we need is a 
methodology, and not a method. A methodology allows for the use of several different 
methods of analysis depending on the nature of the technical system, the type of 
modification sought, and the relevant costs and benefits. Thus, despite the shortcomings of 
the 4-M method, we must not abandon the pursuit of cost-benefit analysis, or return to the 
arbitrary decision making of the past. Instead, we must evolve towards the adoption of a 
methodology such as Costs, Risks, and Values Evaluation (CRAVE). 
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GSWT INTEGRATION PROCESS 


0 GROUNDRULES 

0 ONLY CATEGORY 2, 2E, OR 2S ESR'S ARE CONSIDERED 

0 TECHN ICA L MERI T IS T H E ON LY CRITERIA USED TO EVALLJAT F ESR'S 
T HROUGHOUT TH E IN TEGRATION PR OCESS 

0 THE INTEGRATION PROCESS STARTS WITH THE TEAM LI ST. 

0 THE GSWT TEAM CONSISTING OF A NASA & LSOC SE AND AN SDE 
FROM GSDE PRIORITIZE THE ESR'S WITHIN THEIR SYSTEM BY 
TECHNICAL MERIT AND FISCAL YEAR. 

0 THE NEXT LEVEL OF INTEGRATION (CALLED THE TV LIST1 OCCURS 

ONCE A YEAR. 

0 THE GSWT STEERING COMMITTEE (S/C) REPRESENTATIVES FROM 
TV, 1 7-01 , AND 1 5-01 TAKE THE TEAM LISTS FROM THEIR 
RESPECTIVE ORGANIZATIONS AND INTEGRATE THESE ESR'S INTO A 
SINGLE LIST. 


^Lockheed 

Space Operations Company ^ 


ATTACHMENT *.i 
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GSWT INTEGRATION PROCESS 


0 THE PROCESS WORKS AS FOLLOWS; 

0 THE #1 PRIORITY ESR FROM EACH TEAM'S LIST ARE LAID ON THE 
TABLE. THIS IS A TOTAL OF 22 ESR'S. 

0 THE ESR'S ARE GROUPED BY P-CUT AN INDEX ASSIGNED TO THE 
ESR BY THE GSWT TEAM. IT INDICATES THE SEVERITY OF THE 
IMPACT OF NOT DOING THE REQUESTED MODIFICATION. 

0 USING THE P-CUT, THEIR UNDERSTANDING OF THE PROBLEM 
CONDITION AND THEIR EXPERIENCE AS A GUIDE, THE THREE S/C 
REP'S SELECT WHAT THEY CONSIDER TO BE THE MOST 
IMPORTANT ESR FROM THE ITEMS ON THE TABLE. 

0 THE #2 PRIORITY FROM THAT TEAM (WHOSE ESR WAS SELECTED 
AS MOST IMPORTANT) IS THEN BROUGHT UP ON THE TABLE AND 
COMPETES AGAINST THE OTHER 21 ESR ALREADY ON THE TABLE. 

0 THIS PROCESS CONTINUES UNTIL A SET NUMBER OF ITEMS HAVE 
BEEN SELECTED. THIS IS THE TV LIST 


8 - 19-93 



^^Lockheed 

Space Operations Company 

o 


2 
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GSWT INTEGRATION PROCESS 


0 THE TV LIST IS THEN SUBMITTED TO THE ENGINEERING DIRECTORS 
FROM 1 7-01 AND1 5-01 ,THE LSOC CHIEF ENGINEER, THE TV 
ENGINEERING DIRECTOR, AND ALL TV DIVISION CHIEFS. 

0 A MEETING IS CONVENED TO ALLOW REDLINES TO THE LIST TO BE 
DISCUSSED AND INCORPORATED. 

0 DURING THIS TIME, THE SUPPORT OPS AND TE ORGANIZATIONS ARE 
PERFORMING THE SAME INTEGRATION FOR ESR'S FROM THEIR 
TEAMS USING A SIMILAR PROCESS. THEIR EFFORT COMPILES THE TE 

LIS T. — 

0 THE LAST STEP IN THE INTEGRATION IS PERFORMED BY THE GSWT 
S/C INTEGRATION TEAM AND RESULTS IN THE FYXX GSWT 
INTEGRATED PRIORITY LIST. 

0 THIS TEAM CONSISTS OF THE S/C REPRESENTATIVES FROM TV TE 
30-0, AND 17-01 (ALSO REPRESENTING 15-01). THE S/C CHAIRMAN ’ 
ATTENDS AND ACTS AS A TIE BREAKER IF REQUIRED. 


8 - 19-93 


'Lockheed 

Space Operations Company 



GSWT INTEGRATION PROCESS 


■ 


0 THE TV AND TE LISTS ARE USED AT THIS LEVEL AND THE 

INTEGRATION PROCESS IS THE SAME AS EARILER EXCEPT THERE 
ARE ONLY 2 ESR'S ON THE TABLE AT ANY GIVEN TIME. 

0 THE NUMBER OF ITEMS TO COMPRISE THE LIST IS DETERMINED BY 
THE TEAM AT THE START OF THE PROCESS AND THE INTEGRATION 
CONTINUES UNTIL THIS NUMBER IS REACHED. 

0 WHEN THE PROCESS IS COMPLETE THE LIST IS DISTRIBUTED BY THE 
STEERING COMMITTEE CHAIRMAN. 


L ockheed 

Space Operations Company 
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GSWT INTEGRATION PROCESS 


( 



TV LIST 


8/20/93 




FYXX GSWT INTEGRATED PRIORITY LI ST 



MGMT 


/ 

C Of F 


l\\ 

R&O ASA 


LVL IV GSCCB 


OVER J260K 



u> 
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ATTACHMENT A-2 31 


GSWT STEERING COMMITTEE - PRIORITY DEFINITIONS 

REQUIRED ITEMS THAT ARE: 

P1 _1 _ Requirements dictated by drawing, specification, or directive !o support a 
specific mission eifecilvity 

PI-2 - Modifications required lo support 3 specific flight sysiam/puy oo req ' e 
merit dictated by drawing, specification, or dlrectWa ... 

Pl-a - Modifications which prevent potential loss of !Uo. savere Injury, or signtflc in 
dalg.'lo MglH todv-sr, no lyocadura. o, optional .vo-Korouno. 

PI-4 Modifications which prevent slgnlftcr.nl (>* nrs) processing impact 


REQUIRED ITEMS THAT: 

pn i _ M G et upciraded design reouirnments (with hr.inrdous Implications) 

nX - cTreef design dellclcncles in pHmn.y system Hazardous impllcllona 

REQUIRED iTEMS THAT: 

Pj _1 _ pi orP 2 IlclTTwhete workaround is possible 
P 3_2 — Cancel design dellcronctes in backup systems 
P3--2 - Meetupgindcd tequiroments 

P3 j. - Correct design or documentation inconsisienc.es , , r| laht 

£Q- Reduce (ho potential of injury lo personnel or potemla I 

hardware where a procedural or operational worxjround -x,_.s. 

DESIRABLE ITEMS THAT ARE NEEDED TO: 

P4_i — Prevent obsolf;scencr-ccused problems _ fl , 

P4— 7 - Enhance operations with sliort-lerm. pnyboicx ", , ' ' 

prr-3 -improve processing operations to reduce schedule Usk, 

DESIRABLE ITEMS WITH HO QUANTIFIABLE EFFECT on PROCESS. NG 3U I A/IL-. 

pq-1 - Creato improved practices or efficiencies „ . 

p% \. Enhnnca process ng with long-term payback (more than . yenO 
P pl - Upg^de-sVtems lo align: with current Government regulation* 
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^typrfroy F . White, Choirman 
•-GSW7 electing Committee 
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ATTACHMENT A-3 


LOCKHEED SPACE OPERATIONS COMPANY 
INTERDEPARTMENTAL COMMUNICATION 

DATE: March 29, 19 9^ 





CONTROL 

NUMBER: 3201-94-049 

TO: 

Oi szribution 

OEPT. 

NO. 

BLOG. 

NO. 

MAIL 

CQOE 

FROM: 

J. F. White 

DEPT. 

NO. 32-01 

3LQG. 

NO. LST-2 

•MAIL 

COOE LS0-042 PHONE 


SUBJECT: REPLACEMENT OF THE P-CUT OURING ESR PRIORITIZATION 


Currently a P-CUT is assignee to each ESR as it is entered in CMOS . The P-CUT 
is useful to the Steering Committee Integration Team during the yearly 
prioritization process, out has not been helpful to tne GSWT's in prioritizing 
the ESRs on their individual team lists. 

The GSWT Steering Committee is introducing a new methodology to use in 
determining the priority of EoRs within the GSWT system that will replace the 
P-c'Ji. i hi s new system called “the 4-matrix method" is simoler and more 
straight forward and should benefit your GSWT in evaluating where an ESR should 
nt on your respective team lists. A copy of the method is attached and a 
brief description of each major matrix follows: 

Frequency of Qoeration: 


Each _£3R has a problem condition or event that is described in Block 16 of the 
ESR form. This problem condition is normally the result of the performance of 
some operation such as a particular sequence of an OMI or WAC. This matrix 
correlates the number of times this ooeratl on occurs per flow or month to a 
numerical value. The more frequent the occurrence the higner the value. 

Processing Impact/Tmorovement : 

ihis matrix equates the Block 16 prodem to a Ve.nicle processing impact. This 
impact should be a direct result of the problem condition, a "single point 
lailure" scenario rather than relying on a series of failures to cause a worst 
case situation. 

Likelihood of Occurrence : 

ihis matrix concerns »he probability of the orooiem conciti cn or event 
occurring within a given ti me f r ame . A higner potential of occurrence is 
assigned a higner value in the matrix. 


j we* m 
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TO: Distribution 


3201-94-049 


Cost Assessment : 

The first step in performing this matrix is to fill out the cSR cvaluation-Cost 
impact sheet. This data is similar to the ROM data that has been required in 
the past. When the sheet is complete, you will have calculated the months to 
pay back required to perform the "Cost Assessment" matrix. On the "Cost 
Assessment" matrix, select the appropriate pay back period and numerical value 
associated with the pay back period. 


Multiply the numerical values obtained from the rour matrices together to 
obtain a score for the ESR under evaluation. This score, when compared to the 
scores on other ESRs on your list, should help you understand where this £SR 
should be prioritized. 


Please implement this system immediately. Beginning with the upcoming ri.o 
Integrated Priority Process starting in April of this year, at least the top 
three priorities on your team list need to have been evaluated and selected 
using the 4-matrix method. (Retain the work sheets for these three tSRs to 
provide background data to support your priority). This process is to be used 
on all new ESRs. Use the attached sheets to perform this evaluation. Update 
the CMOS P-CUT field for these first three with the numerical score obtained 
from the matrices. In the summer, the GSWT data tie Ids in CMOS will oe 
revised to allow you to input this information directly into CMOS. Also, SRI 
BM-310 ( 2 ) K is in the process of being revised to reflect this Change. In the 
interim, make the changes to your team list using the attached informat 1 . on. .i 
you have any questions, contact your GSW i Steering Committee Representati ve . 



JFW: 1m 

Attachments: A/S 
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ESR ASSESSMENT CRITERIA 
FREQUENCY OF OPERATION 

5 Multiple Times Per Row/Month 

4 Once Per Flow/Month - Critical Path 

3 Once Per Flow/Month - Non Critical Path 

2 1 - 5/0 M DP/2 Years 

1 Contingency Operation Only 

PROCESSING IMPACT/IMPROVEMENT 

5 More Then 24 Hour Serial Flow Time 

4 More Then 8 Less Then 24 Hour Serial Flow Time 

3 Less Then 8 Hour Serial Flow Time 

2 More Then 24 Hour Parallel Flow Time 

1 Less Then 24 Hour Parallel Flow Time 

LIKELIHOOD OF OCCURRENCE 

4 Expected To Happen Within The Next Flow/Month 

3 Could Happen Within The Next F!ows/6 Months 

2 Could Happen Within Next 1 2 Flows/1 8 Months 

1 Not Expected To Happen In The Next 16 Fiows/2 Years 

COST ASSESSMENT (SCORE FROM REVERSE SIDE) 

5 Payback Within 3 Months 

4 Payback Between 3 and 6 Months 

3 Payback Between 6 and 1 2 Months 

2 Payback Between 1 and 2 Years 
1 Payback Longer Then 2 Years 


TOTAL 


Determine the appropriate classification of the ESR within each cateaory. 
Multiply the score of each category together t 0 determine the ^PP ration . ~Th e 
rating will be compared with other ESP's to pnoritize next year's modification 
budget. The ratings will be independently verified curing the GS'vVT Integration 
Process. 


ri* „vU- PAee * 
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COST ASSESSMENT WORKSHEET 


COST OF PROPOSAL: (ROM) 

Engineering M.H. @ $40.00 $ 

Material $ 

Labor __ M.H. @ $40.00 $ 

FPC $. 

TOTAL COST $ 


"BENEFIT OF PROPOSAL* 
OPERATIONAL SAVINGS: (ANNUAL) 



(A) 


<_ 


M.H. (Old Way) 


( Consumables (Old) - $. 

($ Energy (Old) - $. 


M.H. (New Way) X $40.00 = 
_ Consumables (New) = 

_ Energy (New) = 


TOTAL 

MAINTENANCE SAVINGS: (ANNUAL) 

(P.M. MH.s/Yr (Old) - P.M. M.H.s/Yr (New) X $40.00 = 


(P.M. Mat’l (Old) 


P.M. Mat'! (New) 


J 


5 . 

$. 

$. 


Corrective M.H. (Old) 
(Corrective Mat" (Old) 


- Corrective M.H. (New) 


- Correctible Mat'l (New) 


X $40.00 = $. 

) = s. 


.(B) 


TOTAL (C) 

OTHER SAVINGS (ANNUAL) $ (D) 

PAYBACK: 

A 

P = X 12 = Months To Payback 

B + C + D 

If savings are negative for any factor, add total algebraically. 


To be used with E5R Assessment Matrix 
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Comments on Selected ESRs from the 


Appendix B 


Costs, Risks, and Values Evaluation (CRAVE) Perspective 

by 

Praf Jogiekar 


Costs, Risks- and Values Evaluation (CRAVE) is a methodology. Tnac is, ir is a way of 
thinking — armed with certain fundamental principles and concepts — and a way of 
analyzing available decision alternatives using siniadonaily appropriate techniques and tools 
from a large body of methods consistent with the fundamental principles and concepts. A 
methodology focuses on both the techniques and the processes used in relevant decision- 
making. 

Consequently, the first step in applying CRAVE to the problem of ESR rank-ordering is 
to understand the complexity, diversity, and situational specificity of the ESRs in terms 
of 

a) the magni tudes of costs, risks, and benefits involved, 

b) the degree to which these magnitudes could be quandtatively measured, 

c) the degree to which related decisions affect an ESR's costs and benefits, 

d) the degree to which sub-decision elements (such as the timing of implementation or 
availability of quantity discounts) affect the pertinent costs and benefits, and 

e) any other critical elements. 

Only when we understand rhis complexity and diversity can we create a suitable adaptation of 
the CRAVE methodology to ESR prioritization- Towards that end, I studied several ESRs in 
-re-ms or the modifications they were proposing, their estimated costs, risks, benefits, ESR 
scores, payback periods, etc. I talked to several lead engineers, ESR champions, steering 
committee members, accounting professionals, and third parties familiar with specific ESRs 
and/or familiar with the GSWT process. 

The following comments reflect my informed assessments of the selected ESRs, as well 
as my observations about the ESR prioritization process. Some readers may find these 
comments too cryptic, but of aecsssity, I have assumed that readers are reasonably familiar 
with the selected ESRs and their respective 4-M assessments. 
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K14453. "New Hydraulic Pump Units” 


1. An ESR that does not belong. 

In terms of costs, this ESR stands out as the most important one to analyze carefully. Insofar 
as its estimated costs ($3.8M for Option A and $2.8M for Option Al) exceed the combined 
costs of all 16 remaining ESR s on TV’s FY95 priority list (See Figure 4), it is unfair and 
irrational to treat this as just another ESR to be rank-ordered as Priority 1, or 4 (as the 
Steering Committee did), or 17. Thus, in my opinion this ESR does not belong to the GSWT 
prioritization process. If a comparison with other ESRs must be done, the appropriate 
question would be whether or not the benefits of this ESR exceed the sum total of the benefits 
of all re mainin g ESRs. But such a comparison would be similar to asking whether the value 
of buying a house exceeds the value of feeding and clothing ourselves for several years. With 
that kind of a comparison, we would never buy a house! 

In our personal Eves, although some sacrifices in the amounts we spend on food and clothing 
are necessary to buy a house, through devices such as a mortgage, we manage to buy a house 
without starving. Similarly, organizations must find ways to spend on capital equipment 
without jeopardizing their short-term survival. The best way to do that is to set aside two 
separate funds for capital expenditures and operational expenditures, and to avoid any 
comparison of a project in the capital expenditure category with projects in the operational 
expenditure category. 

At KSC, we do have different funds and funding mechanisms to take care of capital versus 
operational items. Unfortunately, in the GSWT prioritization process, we have not 
stopped comparing capital ESRs with operational ESRs. 

2. Let them fight with their equals. 

If at all, I would have compared this ESR with others costing $1M or more. 

3. Widely different cost estimates. 

As I dug deeper into this ESR, I came across several different cost estimates ranging from 
$1.8M to 3.8M without much explanation for why the estimates were so widely different. I 
was told not to be surprised if it ends up costing $5M by the time it is actually implemented. 
More importantly, it seems that with the accounting system at K.SC, even in retrospect, we 
may never find out how much an ESR really costs. The most disturbing thing for me is the 
fact that the GSWT Steering Committee concerns itself neither with the magnitude of the cost 
estimates nor with the reliability of those estimates. Finally, I am disturbed to note that 
KSC culture is so oblivious to cost consideration. 
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4. If I were King: 

If I were King, considering that 

a) this ESR offers a payback from its quantified benefits (from savings in reduced PRs 
and reduced O&M expenses) within six to eisht years, 

b) additionally, it offers intangible benefits such as avoidance of flight-critical delays 

reduction of noise, and reduction of electric shock hazard, that have not been 
quantified, ' ‘ 

c) the current equipment is 17 years old and it must be replaced within the next two or 
three years, and 

d) proper timing (coordinated with the construction of Fluid Support Stations in the two 
OP? bays) is very important in the case of this ESR, and can save as much as $1M 

I would have approved this ESR for FY94. Perhaps it is still not too late ' 


K15040. PGHM Anti-Free Wheel Mod 

1. Disguised motivation. 

It seems that the GSWT team's real motivation is to modernize and automate PGHM 
operations — a very desirable goal. However, knowing the KSC culture of approving 
primarily (perhaps only) safety related ESRs, the champions of this ESR have framed its 
justification in terms of avoiding potential damage to cryogenic or hypergolic payloads. The 
tact is that given the precautions we take and the manpower we allocate to PGHM operations, 
e likelihood of Z wheel operations causing any major damage to payload is one in several 


2. Inconsistent cost and payback estimates. 

Although an October 1993 study estimates the cost of this ESR to be S49K, the ESR score 
calculations are based on a cost of S108K. No one on the GSWT Steering Committee is 
charged specifically with the responsibility to identify and correct such inconsistencies 
indeed, the steering committee does not know how much expenditure it is authorizing bv 
putting an ESR on its priority list ° J 

Similarly, the GSWT team seems to have made a mistake in reporting that this ESRs payback 

as 11.6 months when they meant 11 .6 years! Once again, the mistake went undetected by the 
steering committee. 

3. Bite the bullet or forget it! 

I am not an engineer, but it seems to me that if we are going to spend S108K, we ought to 

attain a much greater degree of automation than replacing a ball and screw mechanism with a 
machine screw! 
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K15813. EPDM Cover Installation Tool 


1. The steering committee does catch some mistakes. 

Instead of multiplying, the GSWT team had added the scores on the four matrices. The 
steering committee corrected that mistake. The steering committee also seems to have 
challenged the team's claim of a rating of 4 on the Processing Impact factor and corrected it to 

a rating of 2. 

2. Likelihood factor incorrectly rated. 

On the other hand, the steering committee seems to have missed the fact that the GSWT team 
interpreted this factor to mean "likelihood that the operation will be performed" rather than 
"likelihood that a mishap or problem condition would occur." 

3. Operational savings overestimated. 

A reality check suggests that the estimated saving of eight man-hours per EPDM cover 
installation exceeds the actual man-hours currently required for this operation! A payback 
period of four years may be more accurate. Of course, that is sufficient considering the 
reduction in the risk of damaging the covers — a benefit not quantified m the payback 

calculation. 


K153991 . Main Door Windlocks Pin Insertion 

1. Isn't there a cheaper way? 

While it makes sense to avoid a broomstick approach (particularly in the Space Program), 
certainly the replacement should not cost $28K_ 

2. Look for economies of scale. 

This ESR highlights the need to make certain changes in Pad A and Pad B configurations 
simultaneously rather than one at a time. Had we done that, the incremental cost of the 
device for Pad A could have been of the order of $5K rather than $28K. 


K11794. An Additional Winch to Recover from ESPIECP Platform Tilt 

1. Counting benefits that are impossible to obtain through the proposed ESR. 

This ESR estimates the payback period for its investment to be .067 months (or two days)! 
To me this ESR represents the case of a carelessly exaggerated claims. An assumption 
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underlying this claim is that by recovering from a tilt of the ESP/ECP platform we will save 
the cost of an SSME ($60M). The fact is, the damage due to the tilt would be already done 
and straightening out the platform will not in itself save the SSME. We will have to undertake 
all the necessary repairs, and tests as well. If having an additional winch would prevent a 
potential tilt, then we could count the avoided damage times the probability that the damage 
would occur in the absence of the new winch as a saving due to this ESR. 

Another implied assumption in this payback calculation is that a ESP/ECP platform tilt will 
occur with certainty. Although with aging winches, the probability of a tilt occurring is 
increasing, I would say that it is still of the order of one in a hundred. 

Yes, we need to refurbish our aging winches, but this is hardly a rational way of justifying 
that. I am most surprised that the steering committee did not challenge these payback 
calculations. 

2. Inadequacy of the cost assessment worksheet 

It is clear that 4-M's design of the Cost Assessment Worksheet cannot accommodate the 
capture of all relevant risk and benefits associated with this ESR. 

3. Why insist on an exact duplicate of existing winches? 

I am told that for the S340K cost of this ESR, instead of buying one winch that duplicates 
existing winches in form and function, one could easily buy four commercial off-the-shelf 
(COTS) winches that incorporate better technology and avoids single point failures. 

Such a purchase of four new winches would allow us 

a) to use them together for the safest hoisting of crucial loads, and 

b) to disassemble, inspect, and refurbish at least ten of the existing twelve winches, 
with parts to spare from the other two. 

4. The option generating nature of CRAVE. 

The suggestion in Comment 3 above shows that the CRAVE methodology not only helps in 
the assessment of a given ESR, it has the potential to generate better options. 


K 1578 3. KZJ -Band DEMOD Replacement. 

1. Internal inconsistency. 

On the one hand, this ESR claims a processing improvement of more than 24 hours of serial 
flow time. On the other hand, it says that there will be no monetary payback. This seems 
inconsistent. Surely there must be a quantifiable lower bound on the value of a 24-hour serial 
flow improvement. 
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2. The importance of establishing upper and lower bounds on the values of certain 
types of benefits. 

It seems important that a trained cost-benefit analyst should establish upper and lower bounds 
for the value of the various categories in the Processing Impact/Improvement factor. This is a 
very doable task, and it should help us in more rigorous and quantitative assessments of many 
ESRs. 

3. Work around possible. 

As it turns out, this ESR will not contribute to an improvement of 24 hours of serial flow 
time. This is because when the DEMOD does not work, we do have a way to communicate 
between firing room and OPF. The trouble is that communication involves routing through 
JSC and satellite, etc. Commercially, such a communication could cost as much as $100 per 
minute. Luckily, NASA does not have to pay anybody for this communication. However, 
we may be impeding or slowing down other vital communication that truly needs the satellite 
system. Considering all these factors and the fact its estimated cost is $305K, I found this 
ESR to be the most difficult one to decide on. 


K15505. Additional HPD Platforms 

1. We must develop a simpler and speedier low cost mod (LCM) process. 

This is one of the least cosdy ($4K) of the ESRs I studied. It seems unfair to burden such 
low cost proposals with the same degree of rigor of assessment and rank-ordering as ESRs 
costing ten or a hundred times its cost. 

2. Estimate minimum benefits in quantitative terms. 

Although this ESR made no attempt to calculate its payback period, it would have been easy 
to obtain a lower bound on the benefits and an upper bound on that payback. Given that two 
injuries have occurred during the last two years, the average cost of these injuries can be seen 
as the minimum savings brought by this ESR. Of course, one of my presumptions is that we 
have reasonable records from Personnel Department to be able to estimate the costs of those 
injuries. 

3. If I must rank-order the sample of ESRs I have studied: 

Considering the facts that 

a) we have already spent the money for the design of the platforms, 

b) the incremental costs of the remaining work are so small, and 



308 


i i i 


42 


c) it will avoid the types of injuries to workers that have occurred twice during the 
last two years, 

this ESR would be my Priority #1. 

4. Why re-evaluate? 

What I do not understand is why this ESR is being re-evaluated at this stage. The original 
version of this ESR calling for the design and implementation was estimated to cost S35K, 
and we have already done the design work. I would have assumed that implementation of 
that design would not be a new decision point! 


15836. Replace Diver Operated Plug (DOP) sin 4 

1. Reasonably quantifiable costs, risks, and benefits. 

This a good example of ESRs whose costs, risks, and benefits are reasonably quantifiable. 
The use of the 4-M ESR score in such cases would be totally inappropriate. The sole 
criterion for rank-ordering among these ESRs should be the payback period — the shorter the 
better. 

2. Benefits overestimated. 

This ESR claims a payback in seven months by estimating an annual saving of $103K when 
the cost of a new DOP is only $58K. Based on available information, I would re-estimate the 
benefits as below: 

The last time DOP s/n 4 failed was two years back. So let us assume that its probability of 
failure in a year is 0.5. In the event of a failure, we would spend 


One day of sea labor 

$7,700 

One day of sea boat rental 

5,000 

PR Generation & Disposition 

1,600 

Materials & Labor to Fix DOP s/n 4 

19.800 


$34,100 


Thus, assuming that a new DOP will have zero probability of failure, the purchase of a new 
DOP will help us avoid a cost of $34K with a probability of 0.5. Hence, the expected savings 
are (,5)(34K) = 17K per year. This gives us a payback of 41 months. 

To the extent that the probability of DOP s/n 4 f ailin g goes up from year to year, the payback 
period may be shorter than my calculation indicates. But certainly it is not as short as seven 
months. Of course, as I have indicated elsewhere, if the payback is four years or less, I 
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consider an ESR to be desirable. Such ESRs should be prioritized in the increasing order of 
their (correctly estimated) payback periods. 

3. Look for economies of scale. 

Here is another example illustrating the need to take advantage of economies of scale. We 
will need one more DOP within a year or two. It would be considerably cheaper if we 
ordered two at a time, rather than one now and one a year or two later. 

4 . The need for a qualified analyst to assist GSWT Steering Committee. 

I do not expect a steering committee to be able to do the kind of reassessment of benefits I 
have done in Comment 2 above. I believe that if we are serious about incorporating cost- 
effectiveness considerations in our ESR prioritization, the steering committee must be 
assisted by a properly trained and impartial analyst whose charge is to challenge the GS^SVT 
teams' logic and the numbers on each ESR. 


K15818. Digital Control Cards for LOX Pumps 

1. An ESR we cannot afford not to invest in. 

This ESR proposes to spend $46K in order to save three cold flows (LOX) per year, worth 
$25K per flow. Assuming these numbers are correct, KSC must take $46K from its liquid 
oxygen (LOX) budget to fund this ESR immediately, so that we will save $29K this year and 
we will have $75K extra in future years for other desirable projects. 

2. Use the right funding source. 

As I see it, this ESR should be funded from the LOX budget. It should not compete with 
other ESRs for other sources of funds. Any comparison of this ESR with other ESRs would 
be unfair to both. 


KJ4515. Prime Backup HGDS Infinite Zero Gas Source 

1. Not on GSWT steering committee's priority list, very correctly. 

This ESR is not one of the Top 17 on TV’s list, but it is on the "open ESR" database. Yet, I 
have included it in this review primarily because I want to make the following points. 

2. Reasonably quantifiable costs and benefits. 

From the data on this ESR, it seems clear that the costs, risks, and benefits are reasonably 
quantifiable, and hence payback ought to be the sole criterion for this ESRs rank-ordering. 
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An ESR that deserves to be rejected once and for all, dosed, and deleted from 
the database. 


This ESR proposes to spend S118K to be able to use facility GN2 gas in place of ultra pure 
GN2 in everyday standby purge operations, so as to obtain an estimated savings of $7K per 
year. There are no other intangible benefits associated with this ESR. Thus, this ESR will 
take 16 years to payback! 

I believe that such ESRs ought to be rejected once and for all, closed, and deleted from the 
open ESR database" so that the GSWT steering committee is not burdened with its 
consideration year after year, and our computer systems are not burdened with carrying 
unnecessary data. 
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